Add video input file support #260

arnodirlam · 2025-06-25T15:19:56Z

What this does

Enables RubyLLM to recognize and handle video files as attachments, in addition to existing support for images, audio, PDFs, and text.

Why?

Modern LLMs and AI applications increasingly support video input, making it important for RubyLLM to handle video files natively.
This enhancement improves the flexibility and completeness of RubyLLM for users who need to process or analyze video content.

Type of change

Scope check

I read the Contributing Guide
This aligns with RubyLLM's focus on LLM communication
This isn't application-specific logic that belongs in user code
This benefits most users, not just my specific use case

Quality check

I ran overcommit --install and all hooks pass
I tested my changes thoroughly
I updated documentation if needed
I didn't modify auto-generated files manually (models.json, aliases.json)

API changes

Breaking change
New public methods/classes
Changed method signatures
No API changes

Related issues

Fixes #259

arnodirlam · 2025-06-25T15:21:06Z

I've only tested this with Gemini 1.5 Flash and it works.

Updating the models and running the tests makes a lot of errors fail.

What's the best way to proceed from here?

crmne

Hey @arnodirlam I'd love to add this to RubyLLM. Do you have the means to test things out with API keys? There's one little thing to change too.

crmne · 2025-07-16T14:18:49Z

spec/ruby_llm/active_record/acts_as_attachment_spec.rb

Should rather be in chat_content_spec.rb.

Done in 228ab17.

crmne · 2025-07-16T14:20:10Z

Oh and check off the items that you did in the quality check section of the PR. All are required.

…; Update capabilities

arnodirlam · 2025-07-18T16:15:37Z

I've addressed the remaining points now.

~~One thing left: vision is ambiguous now, because it sometimes implies image and video, but in reality only implies image. In fact, only Google Gemini models have video support.~~

~~Should I add methods supports_image? and supports_video? and soft-deprecate supports_vision??~~

EDIT: Just realized vision implies image and pdf in all cases, so vision is totally appropriate.

…pport

arnodirlam · 2025-07-20T12:59:19Z

Added a supports_video? helper ad clarified some comments and docs. This should be good to merge now 👍

crmne

Pretty nice! Just minor changes.

crmne · 2025-08-13T10:38:21Z

README.md

@@ -43,11 +43,12 @@ RubyLLM fixes all that. One beautiful API for everything. One consistent format.

 ```ruby
 # Just ask questions
-chat = RubyLLM.chat
+chat = RubyLLM.chat(model: "gemini-2.0-flash")


let's not change this.

crmne · 2025-08-13T10:38:45Z

README.md

 chat.ask "What's in this image?", with: "ruby_conf.jpg"
+chat.ask "What's happening in this video?", with: "presentation.mp4"


let's call the example video video.mp4

crmne · 2025-08-13T10:39:11Z

docs/guides/chat.md

+
+```ruby
+# Ask about a local video file
+chat = RubyLLM.chat(model: 'gemini-2.0-flash')


change the video model to gemini-2.5-flash

crmne · 2025-08-13T10:39:24Z

docs/index.md

@@ -69,11 +69,12 @@ RubyLLM fixes all that. One beautiful API for everything. One consistent format.

 ```ruby
 # Just ask questions
-chat = RubyLLM.chat
+chat = RubyLLM.chat(model: "gemini-2.0-flash")


let's not change this.

crmne · 2025-08-13T10:39:35Z

docs/index.md

 chat.ask "What's in this image?", with: "ruby_conf.jpg"
+chat.ask "What's happening in this video?", with: "presentation.mp4"


video.mp4

crmne · 2025-08-13T10:41:53Z

spec/spec_helper.rb

@@ -157,6 +157,10 @@
  { provider: :ollama, model: 'qwen3' }
 ].freeze

+VIDEO_MODELS = [
+  { provider: :gemini, model: 'gemini-2.0-flash' }


gemini-2.5-flash

altxtech · 2025-09-11T16:46:33Z

Has this been abandoned? I would like this feature.

altxtech · 2025-09-11T16:57:54Z

@arnodirlam Thank you for this PR. I'm very interested in this feature and I'm willing to resume work on it.

For the original authors and maintainers: Is a successor to this PR welcome?

arnodirlam · 2025-09-12T10:28:38Z

Thank you for taking over in #405, @altxtech. I hope this can go in very soon then 🙌

@arnodirlam

## What this does Adds video file support to RubyLLM. Supersedes #260, originally authored by @arnodirlam. Thank you for the groundwork. > Maintainers: happy to close this if you prefer waiting for the original PR. ### What changed vs #260 - Rebased on current `main` - Resolved conflicts in README/docs and Gemini capabilities - Addressed PR comment reviews - Included `gemini-2.5-flash` as a video model for tests ## Type of change - [ ] Bug fix - [x] New feature - [ ] Breaking change - [ ] Documentation - [ ] Performance improvement ## Scope check - [x] I read the [Contributing Guide](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md) - [x] This aligns with RubyLLM's focus on **LLM communication** - [x] This isn't application-specific logic that belongs in user code - [x] This benefits most users, not just my specific use case ## Quality check - [x] I ran `overcommit --install` and all hooks pass - [x] I tested my changes thoroughly - [x] For provider changes: Re-recorded VCR cassettes with `bundle exec rake vcr:record[provider_name]` - [x] All tests pass: `bundle exec rspec` - [x] I updated documentation if needed - [x] I didn't modify auto-generated files manually (`models.json`, `aliases.json`) ## API changes - [ ] Breaking change - [ ] New public methods/classes - [ ] Changed method signatures - [x] No API changes ## Related issues Closes #259 --------- Co-authored-by: Arno Dirlam <[email protected]> Co-authored-by: Carmine Paolino <[email protected]>

crmne requested changes Jul 16, 2025

View reviewed changes

crmne marked this pull request as ready for review July 16, 2025 14:20

crmne added the enhancement New feature or request label Jul 16, 2025

arnodirlam added 2 commits July 16, 2025 18:37

Add video file support to attachments

aa6d971

Update docs

9604320

arnodirlam force-pushed the feature/video-file-support branch from c1bbf45 to 9604320 Compare July 16, 2025 16:38

Merge branch 'main' into feature/video-file-support

f8c4655

arnodirlam force-pushed the feature/video-file-support branch from dcd8183 to f8c4655 Compare July 18, 2025 14:14

Add tests to RubyLLM::Chat (instead of RubyLLM::ActiveRecord::ActsAs)…

228ab17

…; Update capabilities

Add models supports_video? helper; Docs: Clarify vision vs video su…

6a45b51

…pport

arnodirlam changed the title ~~Add video file support to attachments~~ Add video input file support Jul 23, 2025

arnodirlam requested a review from crmne July 23, 2025 09:49

crmne requested changes Aug 13, 2025

View reviewed changes

altxtech mentioned this pull request Sep 11, 2025

Add video file support #405

Merged

19 tasks

altxtech added a commit to altxtech/ruby_llm that referenced this pull request Sep 11, 2025

changes as per PR review comment on crmne#260

feeb133

arnodirlam closed this Sep 12, 2025

		chat.ask "What's in this image?", with: "ruby_conf.jpg"
		chat.ask "What's happening in this video?", with: "presentation.mp4"

Uh oh!

Add video input file support #260

Add video input file support #260

Uh oh!

Conversation

arnodirlam commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Type of change

Scope check

Quality check

API changes

Related issues

Uh oh!

arnodirlam commented Jun 25, 2025

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crmne commented Jul 16, 2025

Uh oh!

arnodirlam commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arnodirlam commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

altxtech commented Sep 11, 2025

Uh oh!

altxtech commented Sep 11, 2025

Uh oh!

arnodirlam commented Sep 12, 2025

Uh oh!

Uh oh!

arnodirlam commented Jun 25, 2025 •

edited

Loading

arnodirlam commented Jul 18, 2025 •

edited

Loading

arnodirlam commented Jul 20, 2025 •

edited

Loading